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Appendix A. Methods 


This appendix includes information on the data elements, data preparation, missing data, and analytic models 
used in this study. 


Data elements 


The study used administrative data on exclusionary discipline actions, student background characteristics and 
academic achievement, and local education agency characteristics that the Minnesota Department of Education 
shared with the Regional Educational Laboratory Midwest. The Minnesota Department of Education assigned each 
student a unique, deidentified primary key prior to sharing the data. Discipline data included information about 
specific discipline actions. Student characteristics data included data by school year about student background 
characteristics and academic achievement on standardized tests in math and English language arts. Local 
education agency data included information about the characteristics of local education agencies and whether 
the local education agency was identified by the Minnesota Department of Human Rights (MDHR) for a discipline 
reform plan. The study also accessed publicly available data from the National Center for Education Statistics’ 
Common Core of Data on local education agency locale (U.S. Department of Education, n.d.). 


Data preparation 

The study team prepared the data for analysis in three steps: recording and calculating variables, combining data 
from multiple datasets, and creating a matched sample of local education agencies with a discipline reform plan 
and local education agencies with similar characteristics that were not identified by MDHR and did not create a 
plan. The variables used in the analyses are described in table A1. 
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Table A1. Variables used in analyses 


Weelarle) (=) 
Student characteristic 
Gender 


Race/ethnicity 


Eligible for the National School 
Lunch Program 


English learner student status 
Special education status 
Middle school 

High school 


Any prior discipline actions 


Total prior discipline actions 


Local education agency characteristic 


Local education agency type 


Local education agency size 


Local education agency locale 


Student enrollment 


Percentage of students eligible 
for the National School Lunch 
Program 


Percentage of English learner 
students 


Percentage of students in special 
education 


Average standardized test score 
in math 


Prior-year average standardized 
test score in math 


Average standardized test score 
in English language arts 


Prior-year average standardized 
test score in English language arts 


Percentage of students 
experiencing any discipline action 
Average discipline actions per 
student 


Rate of discipline actions for 
racial/ethnic minority students 


Rate of discipline actions for 
American Indian and Black 
students 
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Data 


source 


MDE 
MDE 


MDE 


MDE 
MDE 
MDE 
MDE 
MDE 


MDE 


MDE 
MDE 


CcD 


MDE 


MDE 


MDE 


MDE 


MDE 


MDE 


MDE 


MDE 


MDE 


MDE 


MDE 


MDE 


Description 


Binary indicator of whether a student is female 


Binary indicators of whether a student was Black, American Indian, Asian, 
Hispanic, Native Hawaiian/Pacific Islander, multiracial, or White 


Binary indicator of whether a student was eligible for the National School 
Lunch Program 


Binary indicator of whether a student was an English learner student 
Binary indicator of whether a student was in special education 
Binary indicator of whether a student was in grade 6, 7, or 8 

Binary indicator of whether a student was in grade 9, 10, 11, or 12 


Binary indicator of whether a student experienced a discipline action in the 
prior school year 


Count of the number of discipline actions experienced by a student in the 
prior school year 


Binary indicator if local education agency type was a charter network 


Binary indicator of whether a local education agency is in the 75th percentile 
or greater in total enrollment 


Binary indicators of whether a local education agency was in a city, rural, 
suburban, or town locale 


Count of the number of students assigned to a local education agency 


Percentage of students in the local education agency who were eligible for the 
National School Lunch Program 


Percentage of students in the local education agency who were English learner 
students 


Percentage of students in the local education agency who received special 
education services 


Average student performance on Minnesota Comprehensive Assessments 
math assessments, centered on the state mean 


Average student performance on Minnesota Comprehensive Assessments 
math assessments, centered on the state mean in the prior year 


Average student performance on Minnesota Comprehensive Assessments 
English language arts assessments, centered on the state mean 


Average student performance on Minnesota Comprehensive Assessments 
English language arts assessments, centered on the state mean in the prior year 


Percentage of students in a local education agency experiencing any discipline 
action 


Average number of discipline actions in a local education agency per student 


Rate of discipline actions in a local education agency with racial/ethnic 
minority students per 100 racial/ethnic minority students 


Rate of discipline actions in a local education agency with American Indian or 
Black students per 100 American Indian or Black students 
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Data 


NFVar-Ve)(=) source Description 
Model element 
Plan status MDHR Binary indicator of whether a local education agency was ever identified by 


Minnesota Department of Human Rights and created a discipline reform plan 


Time MDHR A continuous measure of the time before or since the plan was created, 
centered on zero. School years after the creation of discipline reform plans are 
positive, school years before the creation of the discipline reform plans are 


negative 

Post MDHR Binary indicator of whether the year is after the discipline reform plan was 
created 

Plan status and time MDHR An interaction between plan status and the time variable 

Plan status and post MDHR An interaction of plan status and the post variable that demonstrates the 


effect of creating a discipline reform plan after other factors are adjusted for 


MDE is Minnesota Department of Education. MDHR is Minnesota Department of Human Rights. CCD is the National Center for Education Statistics’ Common 
Core of Data. 
Source: Authors’ compilation. 


First, the study team recoded the data elements. The study team transformed incident-level discipline data by 
collapsing multiple exclusionary discipline action observations about one student to a single student-level 
observation that captured all discipline actions data. This transformation created the following student-level 
measures that were used in the analyses: 


e Binary indicator of whether a student experienced any discipline action. 

e Total number of discipline actions a student experienced. 

e Binary indicator of whether a student experienced any discipline action in the prior year. 
e Total number of discipline actions a student experienced in the prior year. 


This step also involved converting student characteristics into binary indicators for the analyses, as well as 
converting student-level data elements into local education agency—level characteristics by school year. 


Student characteristics created through transforming the provided data elements included: 
e Binary indicator of student identified as a racial/ethnic minority student. 

e Binary indicator of student identified as American Indian or Black. 

The local education agency-level variables created included: 

e Total enrollment of students in the local education agency. 


e Average student performance on the Minnesota Comprehensive Assessments math assessments, centered 
on the state mean. 


e Average student performance on the Minnesota Comprehensive Assessments English language arts 
assessments, centered on the state mean. 


e Average student performance on the Minnesota Comprehensive Assessments math assessments, centered 
on the state mean in the prior year. 


e Average student performance on the Minnesota Comprehensive Assessments English language arts 
assessments, centered on the state mean in the prior year. 
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e Total count of students in the local education agency who were of a racial/ethnic background other than 
White. 


e Total count of students in the local education agency who were American Indian or Black. 
e Total count of students in the local education agency who were White. 


e Percentage of students in the local education agency who were of a racial/ethnic background other than 
White. 


e Percentage of students in the local education agency who were American Indian or Black. 

e Percentage of students in the local education agency who were White. 

e Percentage of students in the local education agency who were eligible for the National School Lunch Program. 
e Percentage of students in the local education agency who were English learner students. 

e Percentage of students in the local education agency who were in special education. 

e Percentage of students in a local education agency experiencing any exclusionary discipline action. 

e Average number of exclusionary discipline actions per student in a local education agency. 


e Rate of exclusionary discipline actions taken in a local education agency with racial/ethnic minority students 
per 100 racial/ethnic minority students. 


e Rate of exclusionary discipline actions taken in a local education agency with American Indian or Black 
students per 100 American Indian or Black students. 


e Rate of discipline actions taken in a local education agency with students in special education per 100 students 
in special education. 


Second, the study team appended datasets from multiple years together to create a single analytic dataset across 
the years included in the study and then merged the transformed data on discipline actions with the data on 
student characteristics. The study team first ensured that records were unique to a student in a local education 
agency in a given year. If a student attended schools in multiple local education agencies in a year, the students 
was assigned to the predominantly attended local education agency. After ensuring datasets on discipline and 
student background characteristic and academic achievement were unique by student, local education agency, 
and year, the study team combined the datasets. 


Third, the study team used the combined dataset to create a comparison group of local education agencies similar 
to the local education agencies with a discipline reform plan. The comparison group comprised students in local 
education agencies chosen through propensity score matching (Rubin, 1997), using a one-to-one nearest-neighbor 
approach without replacement. The study team used the larger analytic dataset for research question 2 and used 
only the dataset for local education agencies with a discipline reform plan and matched comparison local 
education agencies for research question 3. The numbers of students in local education agencies with discipline 
reform plans, comparison local education agencies, and all local education agencies without a discipline reform 
plan, by student characteristics, are shown in table A2. 
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Table A2. Counts of students and local education agencies included in the analyses for research questions 2 
and 3, by student group 


| oyer=] m=Xe [Vler-} do) a) PN) iloyer-] im =Xe [Uler-haley a) 

agencies with Comparison agencies without 

discipline foyer} (ol fXoll ol falema=icelaag) 

Research reform plans education agencies re) Eas 

Characteristic included in analyses question (n = 41) (n = 41) (n = 469) 
Total student enrollment 2,3 495,172 243,113 761,470 
American Indian and Black students 2,3 117,091 28,603 70,095 
All racial/ethnic minority students 2,3 255,783 90,934 202,989 
Students eligible for the National School Lunch Program 3 263,561 96,127 320,002 
English learner students 3 86,047 30,660 62,104 
Students in special education 2,3 84,826 41,237 134,817 
Students ever suspended 2,3 33,955 13,297 38,385 
Students ever excluded 2,3 18 2 29 
Students ever expelled 2,3 84 29 135 


Source: Authors’ analysis of data provided by the Minnesota Department of Education. 


The study team assessed baseline equivalence by examining the effect size of preintervention outcome measures 
for local education agencies with discipline reform plans and for comparison local education agencies. For a quasi- 
experimental study to meet the requirements of the What Works Clearinghouse Group Design Standards with 
Reservations Version 4.1 without the inclusion of a preintervention outcome in the analysis, the effect size of 
baseline differences between an intervention and comparison group must be less than 0.05 (What Works 
Clearinghouse, 2020). If the effect size is 0.05—0.25, the analytic design must include a measure of the 
preintervention outcome to meet standards. The baseline equivalence analysis found differences of 0.077 to 0.082 
standard deviations, indicating that analyses must include measures of preintervention outcomes to have the 
potential of meeting What Works Clearinghouse Group Design Standards With Reservations (table A3). 


Table A3. Baseline equivalence for preintervention measures of discipline outcomes, 2014/15-2016/17 


Students in local Students in 
education agencies comparison 

with discipline KoXer=] m=xe [Ler 1u(o) a) 

reform plans agencies 

(n = 409,751) (n = 198,007) 

S\ilalel-lac! SliJalel-lae! Effect 

Outcome Mean deviation WieeTa) deviation size 
Any exclusionary discipline action experienced by student 0.041 0.198 0.027 0.162 0.077 
Total number of exclusionary discipline actions per student 0.077 0.513 0.042 0.321 0.082 


Source: Authors’ analysis of data provided by Minnesota Department of Education. 


Missing data 

Data provided by the Minnesota Department of Education were considered population data; as a result, statistical 
testing was not necessary for answering research questions 1 and 2. The Minnesota Department of Education 
provided discipline data and student demographic data separately. In total, 49 cases appeared in the discipline 
data for which the student identification number and year combination did not match a student identification 
number and year combination in the student demographic data. These 49 records were excluded from the 
analyses and represented 0.002 percent of the dataset. It is also possible that discipline data were not reported 
to the Minnesota Department of Education. There is no available reference by which this possibility can be 
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checked. The possibility of missing data is a limitation to any potential inferences. No data were imputed for any 
of the analyses. 


Analytic models 


The following section describes the analytic models used to answer the four research questions in this study. 


Research question 1. For research question 1 the study team compared key characteristics of local education 
agencies with discipline reform plans and key characteristics of all local education agencies in Minnesota. The 
characteristics included local education agency locale, average student enrollment, percentage of female 
students, percentages of students by race/ethnicity, percentage of students eligible for the National School Lunch 
Program, percentage of students in special education, percentage of English learner students, average 
standardized test scores in math and English language arts, and prior exclusionary discipline actions. Differences 
of 5 percentage points or greater were considered meaningful. 


Research question 2. For research question 2 the study team calculated the rates of students who experienced an 
exclusionary discipline action, by school year. First, the study team calculated the total number of discipline 
actions experienced by American Indian or Black students, all racial/ethnic minority students, White students, 
students in special education, and students not in special education. Discipline action rates per 100 students were 
calculated separately for local education agencies with discipline reform plans and local education agencies 
without discipline reform plans, by school year. 


The formula used to calculate the exclusionary discipline action rate for a student group in a given school year was 
as follows (where x is the student group of interest and j is the school year of interest): 


Total Enrollment; 
100 


Research question 3. For research question 3 the study team used a multilevel comparative interrupted time- 
series model with a matched comparison group. A comparative interrupted time series is a quasi-experimental 
design that can be used for causal inference when a comparison or control series can be constructed (Shadish et 
al., 2001). This method compares the outcomes of a treatment group with outcomes for a comparison group after 
a treatment occurs, relative to their baseline prior to program implementation, to determine program effect after 
controlling for prior trends. This method is appropriate for contexts in which an abrupt policy change occurs and 
in which preintervention and postintervention data are available. 


Rate, ; = Total Discipline Actions ;/( 


The intervention group comprised students in the 41 local education agencies that created discipline reform plans 
in 2017/18. The comparison group comprised students in 41 similar local education agencies not required to 
create discipline reform plans chosen through propensity score matching (Rubin, 1997). An advantage of using a 
comparative interrupted time-series design with a matched comparison group is that the model is flexible and 
able to account for nonparallel trends in pretreatment outcomes between the intervention and comparison 
groups. 


To construct the comparison group, the study team matched local education agencies with discipline reform plans 
to other local education agencies on their characteristics prior to the creation of discipline reform plans, including 
average student enrollment; the percentage of racial/ethnic minority students; the percentage of students eligible 
for the National School Lunch Program; the percentage of students in special education; agencies’ use of discipline 
actions (the percentage of students with any discipline action, the discipline action rate per 100 students, the 
discipline action rate per 100 students for American Indian or Black students, the discipline action rate per 100 
students for all racial/ethnic minority students, and the discipline action rate per 100 students for students in 
special education); average standardized test score in math; average standardized test score in English language 
arts; whether the local education agency was a traditional district or a charter network, and locale (see table A4). 
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Table A4. Differences between local education agencies with discipline reform plans and comparison agencies 
without discipline reform plans 


|KoYer=] m=Xe | (er-} adel) 


agencies with Comparison 

(ol Yet ol fat=masvcelaan KoXer=] m=xe [Ler 1uloy a) Silalel-ael p4cve| 
|oYor-} =o [or-} dle) aur-}-x-1 plea acolat-la-leuk-la sya (ocm lave Ule(-Yomlamanr-lne allals4 plans average agencies average mean 
analyses (n = 41) (n = 41) difference 
Average student enrollment 7,733.24 3,843.59 —2.47* 
Percent racial/ethnic minority students 56.76 58.60 0.26 
Percent students eligible for the National School Lunch Program 55.69 59.71 0.67 
Percent students in special education 15.65 14.58 -1.07 
Percent students with any discipline action 5.96 6.92 0.76 
Total number of discipline actions per 100 students 10.50 12.82 0.78 
Total number of discipline actions for American Indian and Black 20.02 19.48 —0.15 
students per 100 American Indian and Black students 
Total number of discipline actions for racial minority students 13.89 15.17 0.42 
per 100 racial minority students 
Total number of discipline actions for students in special 11.61 13.78 0.90 
education per 100 students in special education 
Math performance compared with state average —5.89 —4.72 0.60 
English language arts performance compared with state average —5.66 -5.34 0.17 
Percent local education agencies with charter status 19.51 39.02 1.96* 
Town locale 9.76 14.63 0.67 
Suburb locale 41.46 34.15 0.68 
City local 36.59 39.02 0.23 
Rural local 12.20 12.20 <0.00 


* Significant at p < .05; ** significant at p < .01. 
Source: Authors’ analysis of data provided by the Minnesota Department of Education. 


The study team used two similar analytic models to answer research question 3 and examine the robustness of 
the findings. The primary model was a three-level hierarchical linear model that incorporated local education 
agency and cohort-level random intercepts and adjusted for prior trends in exclusionary discipline actions; 
whether a student experienced a discipline action in the prior school year; student background characteristics 
(including gender, race/ethnicity, eligibility for the National School Lunch Program, special education status, 
English learner student status, and grade level); and the characteristics of the local education agency (including 
whether a local education agency was a traditional district or a charter network, total student enrollment, and 
average student performance on standardized tests in math and English language arts in the prior school year). 
This model allowed the study team to examine how discipline outcome changes varied across local education 
agencies. 


The second analytic model was a fixed effects model with cohort fixed effects, local education agency covariates, 
and clustered standard errors accounting for the same student appearing in the dataset over multiple years. The 
model adjusted for the same variables as in the first model. For each outcome the most methodologically 
appropriate model was used based on the distribution of the outcome data (for example, the study team used 
generalized linear models for the total discipline actions per student outcome but used logistic regression models 
for the student experiencing any discipline action outcome). 


The following model was used for identifying the effect on discipline outcomes: 
f Yiye) = BotBiXijt + B2W jt + B3MDHR; + ByTime;, + Bs Post, + BeTime, * MDHR;; + B7Post * MDHR; 
+ vj + Ujt + Ci jt 


where f (Yee) is the outcome of interest and Y;j, is a latent variable connected to the outcome through the 
function f (); f () is the link function appropriate for a given outcome variable (logit for binary outcomes such as 
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whether the student received any discipline action, natural log for the number of discipline actions per student in 
a given year); X;j_ is a vector of student characteristics for student / in district j at time t; Wj, is a vector of local 
education agency characteristics at time t; MDHR; is an indicator of whether a local education agency is part of 
the intervention group; and £3 is the difference in the average outcome between local education agencies with a 
discipline reform plan and comparison local education agencies; 8, is an estimate of change in the outcome from 
year to year, where Time, is equal to zero when the school year is the 2017/18 school year, prior years are 
negative decreasing from zero, and following years are positive increasing from zero; Post;, is an indicator for the 
postintervention time period t (the 2018/19 school year); Bg is a vector showing the change in difference in 
outcomes between local education agencies with discipline reform plans and comparison local education agencies 
at each time t; B7 is a vector showing the change in difference in outcomes between local education agencies with 
discipline reform plans and comparison local education agencies for the 2018/19 school year; and Vj, Ujt, ijt are 
local education agency, time, and individual-level random-error terms. 


This analysis produced estimates of regression coefficients (betas) and standard errors that identified statistically 
significant predictors of outcomes. However, because different link functions were used for different outcomes, 
the interpretation of the regression coefficients varied. For example, when the outcome was binary (whether a 
student experienced any discipline action or not), the link function was logit. The estimates from the analyses 
were reported as predicted probabilities in the report to facilitate interpretation (more detail is included in table 
B5 in appendix B). Estimates were also reported as odds ratios (see table B4); odds ratios greater than 1 indicated 
a positive statistical relationship and those less than 1 indicated a negative relationship. When the outcome was 
a count variable (such as the total number of discipline actions a student experienced in a school year), the link 
function was the natural log, and the regression coefficients were interpreted as the difference between the log 
of expected counts. 


Research question 4. For research question 4 the study team examined the content of the discipline reform plans. 
First, the study team reviewed the discipline reform plans published by MDHR for all 41 local education agencies 
with discipline reform plans, as well as the updates that had been published since fall 2017 (which are released in 
September and February of each year). Plan updates included any changes or shifts that were made to the initial 
plan, as well as progress reports of all previously laid out action steps. To develop the categories for discipline reform 
approaches, two team members noted recurring themes in the plans and updates. Then, the team members 
consolidated related reforms into categories based on the content of each reform, consulting the literature as 
needed. After creating this initial set of categories, the team consulted with the Minnesota Department of Education 
and MDHR to verify the accuracy of the categories. Next, two team members conducted pilot reviews of the same 
three discipline reform plans to test the coding framework and calibrate their ratings. After achieving agreement, 
they coded the remaining discipline reform plans and updates. 
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Appendix B. Supporting analysis 


For research question 1 the study team identified the locale types and locations of local education agencies with 
discipline reform plans. The study team mapped the local education agency locations to display the geographic 
distribution (map B1). 


Map B1. Map of local education agencies identified in 2017 by the Minnesota Department of Human Rights as 
using exclusionary disciplinary practices disproportionately among racial/ethnic minority students and 
students in special education 


Source: Authors’ compilation based on discipline reform plans published on the Minnesota Department of Human Rights’ website. 


For research question 2 the study team calculated the rates of discipline actions for students by year for American 
Indian and Black students, all racial/ethnic minority students, White students, students in special education, and 
students not in special education (table B1). The study team also calculated these rates separately for students in 
local education agencies with discipline reform plans and students in other local education agencies (table B2). 


Table B1. Exclusionary discipline action rates per 100 students among Minnesota public school students, 
2014/15-2018/19 


2014/15 2015/16 2016/17 2017/18 2018/19 Overall 
Student group (n = 818,127) (n=825,433) (n=832,217) (n=837,568) (n=840,012) (n= 1,174,713) 
All students (per 100 students) 4.69 5.18 5.06 5.61 5.36 5.18 
Race/ethnicity 
American Indian and Black students 17.56 20.01 17.91 19.73 17.90 18.63 
All racial/ethnic minority students 9.53 10.66 9.86 10.87 9.93 10.18 
White students 2.48 2.55 2.64 2.87 2.89 2.68 
Special education status 
Students in special education 15.92 16.68 15.63 16.74 15.69 16.14 
Students not in special education 2.80 3.23 3.22 3.62 3.45 3.27 


Note: Discipline action rates are the number of suspensions, exclusions, and expulsions per 100 students. 
Source: Authors’ analysis of data provided by the Minnesota Department of Education. 
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Table B2. Exclusionary discipline action rates per 100 students among students in local education agencies with discipline reform plans and local 
education agencies without plans, 2014/15-2018/19 


(oW-1¢-11 
2014/15 2015/16 2016/17 2017/18 2018/19 (2014/15-2018/19) 
Without i Without i Without i Without i Without i Without 
plan plan re) ETA) plan re) ETa) plan 

Silo (-larmaelt] (n = 313,935) (n = 504,192) (n = 316,102) (n = 509,331) (n = 316,393) (n =515,824) (n =317,063) (n = 520,505) (n=316,479) (n =523,533) (n= 495,172) (n= 761,470) 
All students (per 7.05 3.21 8.33 3.23 7.71 3.43 8.39 3.92 7.50 4.07 7.80 3.58 
100 students) 
Race/ethnicity 
American Indian 20.66 11.13 24.72 10.51 21.10 11.77 23.44 12.83 20.90 12.54 22.17 11.81 
and Black 
students 
All racial/ethnic 11.71 6.41 13.98 6.06 12.31 6.62 13.55 7.41 11.96 7.41 12.70 6.82 
minority students 
White students 2.74 2.37 2.85 2.43 3.07 2.47 3.00 2.81 2.73 2.95 2.88 2.61 
Special education status i 
Students in 25.46 9.84 27.68 9.81 24.63 10.17 26.10 11.13 23.13 11.30 25.38 10.48 
special education 
Students not in 3.90 2.11 5.02 2.11 4.80 2.25 5.26 2.62 4.66 2.72 4.73 2.37 


special education 


Note: Discipline action rates are the number of suspensions, exclusions, and expulsions per 100 students. 
Source: Authors’ analysis of data provided by the Minnesota Department of Education. 


Hierarchical linear model analyses findings 


For research question 3 the study team used two analytic models. Findings from the primary hierarchical linear 
model are in tables B3 and B4. Table B5 shows the predicted probability of a student experiencing any discipline 
action based on enrollment in a local education agency that created a discipline reform plan or in a comparison 
agency that did not create a reform plan, after student and local education agency characteristics and prior trends 
in exclusionary discipline actions were controlled for. Table B6 shows the 95 percent confidence interval for the 
total number of discipline actions a student would experience in a year after student and local education agency 
characteristics and prior trends in exclusionary discipline actions were accounted for. 


Table B3. Impact of creating a discipline reform plan on student exclusionary discipline outcomes in local 
education agencies with a discipline reform plan, hierarchical linear model, 2014/15-2018/19 


Coefficient 
Discipline outcome (Ch-lalel-Inem-lane)p) p-value 
Any exclusionary discipline action (odds ratio) 1.063 .664 
(0.149) 
Total exclusionary discipline actions (count per student) —0.097 753 
(0.310) 


Source: Authors’ analysis of data provided by the Minnesota Department of Education. 
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Table B4. Impact of creating a discipline reform plan on student exclusionary discipline outcomes in local 
education agencies with a discipline reform plan, with student and local education agency covariates, 


hierarchical linear model, 2014/15—2018/19 


WETarle) (=) 


Student predictor 


Any discipline 


EYeid(ola) 


(ole oka g-hale) 
(Gimlalel-lnem-lanela) 


0.01 (0.00)*** 


Total discipline 


actions 


Coefficient 
(Gilalel-laem-lanelp) 


—5.43 (0.05)*** 


Intercept 


Female 


0.52 (0.00)*** 
2.55 (0.02)*** 


—0.63 (0.01)*** 


American Indian or Black students 


2.66 (0.03)*** 


0.75 (0.01)*** 


Eligibility for the National School Lunch Program 


Special education status 


2.49 (0.02)*** 
0.67 (0.01)*** 


0.92 (0.0)*** 
0.97 (0.00)*** 
—0.42 (0.01)*** 


English learner student status 


3.65 (0.04)*** 


1.07 (0.01)*** 


Enrolled in middle school grade 


Enrolled in high school grade 


2.47 (0.02)*** 
4.85 (0.08)*** 


0.74 (0.01)*** 
1.55 (0.01)*** 


Any discipline action in the prior year 


(0.00) 
(0.00) 
(0.02) 
(0.03) 
(0.02) 
(0.01) 
(0.04) 
(0.02) 
(0.08) 
(0.01) 


1.27 (0.01)*** 


( 
( 
( 
( 
(0.00) 
(0.01) 
(0.01) 
(0.01) 
(0.01) 
(0.00) 


0.10 (0.00)*** 


Total discipline actions in the prior year 


Local education agency predictor 


Created discipline reform plan, all years 


0.83 (0.11 


0.36 (0.58 


Treatment effect (created discipline reform plan, postidentification) 


1.06 (0.15 
0.90 (0.09 


—0.10 (0.31 
0.06 (0.04 


After creation of discipline reform plans, all local education agencies 


Time, all local education agencies 


1.00 (0.03 
0.87 (0.03)** 


—0.02 (0.01 
—0.16 (0.07)* 


Time, local education agencies with a discipline reform plan 


( 
( 
( 
1.00 (0.00 


( 
( 
( 
( 
( 
0.00 (0.00 


Average percentage of students eligible for the National School Lunch Program 


Average percentage of students in special education 


** 


1.00 (0.00 


*R 


0.00 (0.00 


Average percentage of students who were English learner students 


Average math performance in prior year 

Average English language arts performance in prior year 
City locale 

Suburb locale 


Town locale 


1.00 (0.00 
1.01 (0.03 
1.00 (0.03 
1.36 (0.27 
1.08 (0.22 
( 
( 


) 
) 
) 
) 
) 
) 
) 
) 
) 
) 
) 
) 
1.07 (0.26) 
) 


aa Ne rrr 
( 
0.64 (0.07)*** 
( 
( 


1.62 (0.50 


0.00 (0.00 
0.01 (0.01 
—0.01 (0.01 
0.65 (0.10)*** 


) 
) 
) 
) 
) 
) 
) 
) 
) 
) 
) 
) 
0.45 (0.08)*** 
) 


0.00 (0.00)** 


Total student enrollment 


Random effect (variance component) 


Variance in cohort (level 2) 


0.14 
0.24 


0.49 
0.80 


Variance in local education agency (level 3) 


* Significant at p < .05, ** significant at p < .01, *** significant at p < .001. 


Source: Authors’ analysis of data provided by the Minnesota Department of Education. 
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Table B5. Marginal predicted probability of experiencing any exclusionary discipline action, by discipline 
reform plan status, after all other model covariates were accounted for, hierarchical linear model, 2014/15- 
2018/19 


Margin 95% confidence interval 
(Gitlatek-lae} 
Discipline reform plan status lagoya) Minimum Maximum 
Local education agencies with a discipline reform plan 0.020 (0.00) .0O 0.016 0.024 
Comparison local education agencies 0.021 (0.00) .0O 0.015 0.027 


Source: Authors’ analysis of data provided by the Minnesota Department of Education. 


Table B6. Change in total number of exclusionary discipline actions, by discipline reform plan status, after all 
other model covariates were accounted for, hierarchical linear model, 2014/15-2018/19 


95% confidence interval 


Discipline reform plan status Minimum Maximum 
Local education agencies with a discipline reform plan —0.705 0.511 
Comparison local education agencies —0.608 0.608 


Source: Authors’ analysis of data provided by the Minnesota Department of Education. 


Fixed effects analyses findings 

To check the robustness of the findings, the study team also analyzed the data using fixed effects models. The 
results from these analyses are in tables B7 and B8. The findings are similar to those of the hierarchical linear 
models analyses. 


Table B7. Impact of creating discipline reform plan on student exclusionary discipline outcomes in local 
education agencies with discipline reform plans, fixed effects model, 2014/15-2018/19 


Coefficient 
Discipline outcome (Gh-Valel-lnem-lanela) p-value 
Any discipline action (odds ratio) 0.988 .67 
(0.028) 
Total discipline actions (count per student) 0.050 12 
(0.032) 


Source: Authors’ analysis of data provided by the Minnesota Department of Education. 


Table B8. Impact of creating a discipline reform plan on student exclusionary discipline outcomes in local 
education agencies with discipline reform plans, with student and local education agency covariates and 


cohort fixed effects, 2014/15-2018/19 


rNaWael ello) iL al=s aKo}ic-] Kel xe o) il al=) 
(ole oka g-hale) Coefficient 

Variable '(ialellaem lane) a) (standard error) 
Student predictor 
Intercept 0.01 (0.00)*** —3.81 (0.03)*** 
Female 0.52 (0.00)*** -0.54 (0.01)*** 
American Indian or Black 2.64 (0.02)*** 0.83 (0.01)*** 
Eligible for the National School Lunch Program 2.69 (0.03)*** 1.00 (0.01)*** 
Special education status 2.43 (0.02)*** 0.95 (0.01)*** 
English learner student status 0.67 (0.01)*** —0.46 (0.01)*** 
Enrolled in middle school grade 3.45 (0.03)*** 1.02 (0.01)*** 
Enrolled in high school grade 2.31 (0.02)*** 0.53 (0.01)*** 
Any discipline action in the prior year 5.00 (0.10)*** 1.58 (0.01)*** 
Total discipline actions in the prior year 1.27 (0.01)*** 0.10 (0.00)*** 
Local education agency predictor 
Created discipline reform plan, all years 0.84 (0.01)*** —0.19 (0.02)*** 
Treatment effect (created discipline reform plan, postidentification) 0.99 (0.03) —0.05 (0.03) 
After creation of discipline reform plans, all local education agencies 0.86 (0.02)*** -0.26 (0.03)*** 
Time, all local education agencies 1.11 (0.01)*** 0.17 (0.01)*** 
Time, local education agencies with a discipline reform plan 0.94 (0.01)*** —0.10 (0.01)*** 
Average percentage of students eligible for the National School Lunch Program 1.00 (0.00)*** 0.00 (0.00) 
Average percentage of students in special education 1.00 (0.00)*** 0.00 (0.00) 
Average percentage of students who were English learner students 1.00 (0.00)*** 0.00 (0.00) 
Average math performance in prior year 1.04 (0.01)*** 0.06 (0.01)*** 
Average English language arts performance in prior year 0.93 (0.01)*** —0.10 (0.01)*** 
City locale 0.83 (0.02)*** -0.27 (0.02)*** 
Suburb locale 0.79 (0.02)*** —0.30 (0.02)*** 
Town locale 0.83 (0.02)*** —0.27 (0.02)*** 
Total student enrollment 1.00 (0.00)*** 0.00 (0.00)*** 


*** Significant at p < .001. 
Source: Authors’ analysis of data provided by Minnesota Department of Education. 


Power analyses findings 


The study team estimated the minimum detectable effect size using PowerUp! (Dong & Maynard, 2013). The 
design is a three-level random effects cluster design in which an average of 5,756 students are nested within 409 
cohorts in 82 local education agencies. A minimum detectable effect size of 0.27 with 80 percent probability was 
calculated using a two-tailed test, at a .05 level of significance. This calculation was based on the assumption that 
90 percent of the variance in the level 1 outcomes, 25 percent in the level 2 outcomes, and 15 percent in the level 
three outcomes are explained by level 1, level 2, and level 3 covariates, and an intraclass correlation coefficient 
of .10 at level 2 and .20 at level 3. 
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